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Amendments to the CLAIMS: 

Without prejudice, this listing of the claims replaces all prior versions and listings of 
the claims in the present application: 

LISTING OF CLAIMS : 

1. (Original) A foreground object and backgroimd sprite separation and extraction 
method for extracting a foreground object and a backgroxmd sprite, comprising the steps of: 

obtaining a global motion for transforming a coordinate system between a reference 
frame and a frame for each of frames in a moving image; 

mapping an original image corresponding to said frame into a reference coordinate 
system for said each of frames by using said global motion, and obtaining a pixel value at a 
point in said reference coordinate system from pixel values of pixels which exist in the same 
point; 

generating a provisional sprite where foreground objects are deleted; 

cutting out a first image from said provisional sprite by using said global motion; 

obtaining a difference image between said first image and said original image; 

extracting a foregroimd object image as a region in said difference image where each 
difference value in the region is equal to or higher than a threshold, and extracting other 
region as a background image; 

mapping said backgroimd image to said reference coordinate system by using said 
global motion for said each of frames by inserting a new pixel in a point where a pixel value 
is not yet decided, or by overwriting a pixel, for generating and outputting said background 
sprite. 

2. (Original) The foreground object and background sprite separation and extraction 
method as claimed in claim 1 , further comprising the steps of: 

cutting out a second image from said background sprite by using said global motion; 

obtaining a difference image between said second image and said original image; 

extracting a foreground object image as a region in said difference image where each 
difference value in the region is equal to or higher than a threshold. 
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3. (Original) A foreground object and background sprite separation and extraction 
apparatus for extracting a foreground object and a background sprite, comprising; 

means for obtaining a global motion for transforming a coordinate system between a 
reference frame and a frame for each of frames in a moving image; 

means for mapping an original image corresponding to said frame into a reference 
coordinate system for said each of frames by using said global motion, and obtaining a pixel 
value at a point in said reference coordinate system from pixel values of pixels which exist in 
the same point; 

means for generating a provisional sprite where foreground objects are deleted; 
means for cutting out a first image from said provisional sprite by using said global 
motion; 

means for obtaining a difference image between said first image and said original 

image; 

means for extracting a foreground object image as a region in said difference image 
where each difference value in the region is equal to or higher than a threshold, and extracting 
other region as a background image; 

means for mapping said background image to said reference coordinate system by 
using said global motion for said each of frames by inserting a new pixel in a point where a 
pixel value is not yet decided, or by overwriting a pixel, for generating and outputting said 
background sprite. 

4. (Original) The foreground object and background sprite separation and extraction 
apparatus as claimed in claim 3, fiuther comprising: 

means for cutting out a second image from said backgrovmd sprite by using said 
global motion; 

means for obtaining a difference image between said second image and said original 

image; 

means for extracting a foreground object image as a region in said difference image 
where each difference value in the region is equal to or higher than a threshold. 
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5. (Original) A computer readable medium storing program code for causing a 
computer to extract a foreground object and a background sprite, comprising: 

program code means for obtaining a global motion for transforming a coordinate 
system between a reference frame and a frame for each of frames in a moving image; 

program code means for mapping an original image corresponding to said frame into a 
reference coordinate system for said each of frames by using said global motion, and 
obtaining a pixel value at a point in said reference coordinate system from pixel values of 
pixels which exist in the same point; 

program code means for generating a provisional sprite where foreground objects are 
deleted; 

program code means for cutting out a first image from said provisional sprite by using 
said global motion; 

program code means for obtaining a difference image between said first image and 
said original image; 

program code means for extracting a foregroxmd object image as a region in said 
difference image where each difference value in the region is equal to or higher than 
threshold, and extracting other region as a background image; 

program code means for mapping said backgroimd image to said reference coordinate 
system by using said global motion for said each of frames by inserting a new pixel in a point 
where a pixel value is not yet decided, or by overwriting a pixel, for generating and outputting 
said background sprite. 

6. (Original) The computer readable medium as claimed in claim 5, further 
comprising: 

program code means for cutting out a second image from said backgroimd sprite by 
using said global motion; 

program code means for obtaining a difference image between said second image and 
said original image; 

program code means for extracting a foregroimd object image as a region in said 
difference image where each difference value in the region is equal to or higher than 
threshold. 
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7. (Original) A segmentation mask extraction method in object coding in moving 
image coding, comprising the steps of: 

receiving a foreground mask image where a foreground part is represented by first 
value and a background part is represented by a second value; 

providing a first value as an alpha value to all shape pixels in each of first 
macro-blocks when the number of pixels of said foreground part in said first macro-block is 
equal to or larger than a fijrst predetermined value n (n_^ 1); 

providing said first value as said alpha value to all shape pixels in each of second 
macroblocks when the mmiber of pixels of said foreground part in said second macro-block is 
equal to or larger than a second predetermined value m (m<n), wherein said second 
macro-block is close to said first macro-block where said first value is provided; and 

outputting said segmentation mask. 

8. (Original) The segmentation mask extraction method as claimed in claim 7, fiirther 
comprising the steps of: 

receiving each of third macro-blocks which has been determined as said background 
part; and 

providing said first value to said third macro-block when a difference image between 
a background image and an original image which correspond to said third macro-block 
includes a pixel which has a difference value equal to or larger than a threshold. 

9. (Original) A segmentation mask extraction method in object coding in moving 
image coding, comprising the steps of: 

receiving a foreground mask image; 

generating a number map by calculating the number of pixels of a foreground part for 
each of macro-blocks in said foreground mask image; 
initializing a foreground map; 

providing a predetermined value to each of positions in said foreground map 
corresponding to first macro-blocks when a value of said number map corresponding to said 
first macro-block is equal to or larger than a first predetermine value n (n^ 1); 
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providing said predetermined value to each of positions in said foreground map 
corresponding to second macro-blocks when a value of said number map corresponding to 
said second macro-block is equal to or larger than a second predetermined value m (m<n), 
wherein said second macro-block is lose to said first macro-block where said predetermined 
value is provided; and 

generating said segmentation mask fi-om said foreground map and outputting said 
segmentation mask. 

10. (Original) A segmentation mask extraction apparatus in object coding in moving 
image coding, comprising: 

means for receiving a foreground mask image where a foreground part is represented 
by a first value and a background part is represented by a second value; 

first macro-block approxunation means for providing a first value as an alpha value to 
all shape pixels in each of first macro-blocks when the number of pixels of said foreground 
part in said first macro-block is equal to or larger than a first predetermined value n (n ^ 1); 

second macro-block approximation means for providing said first value as said alpha 
value to all shape pixels in each of second macro-blocks when the number of pixels of said 
foreground part in said second macro-block is equal to or larger than a second predetermined 
value m (m<n) wherein said second macro-block is close to said first macroblock where said 
first value is provided in said first macro-block approximation means; and 

means for outputting said segmentation mask. 

1 1 . (Original) The segmentation mask extraction apparatus as claimed in claim 10, 
further comprising: 

means for receiving each of third macro-blocks which has been determined a said 
background part; and 

means for providing said first value to said third macro-block when a difference 
image between a backgroimd image and an original image which correspond to said third 
macro-block includes a pixel which has a difference value equal to or larger than a threshold. 
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12. (Original) A segmentation mask extraction apparatus in object coding in moving 
image coding, comprising the steps of: 

means for receiving a foreground mask image; 

means for generating a number map by calculating the number of pixels of a 
foreground part for each of macro-blocks in said foreground mask image; 
means for initializing a foreground map; 

means for providing a predetermined value to each of positions in said foreground 
map corresponding to first macro-blocks when a value of said number map corresponding to 
said first macro-block is equal to or larger than a first predetermined value n (n ^ 1); 

means for providing said predetermined value to each of positions in said foreground 
map corresponding to second macro-block when a value of said number map corresponding 
to said second macroblock is equal to or larger than a second predetermined value m (m<n), 
wherein said second macro-block is close to said first macro-block where said predetermined 
value is provided; and 

generating said segmentation mask fi-om said foreground map and outputting aid 
segmentation mask. 

13. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask in object coding in moving image coding, 
comprising: 

program code means for receiving a foreground mask image where a foreground part 
is represented by a first value and a background part is represented by a second value; 

first macro-block approximation program code means for providing a first value as an 
alpha value to all shape pixels in each of first macro-blocks when the number of pixels of 
said foreground part in said first macro-block is equal to or larger than a first predetermined 
value n^ 1); 

second macro-block approximation program code means for providing said first value 
as said alpha value to all shape pixels in each of second macro-blocks when the number of 
pix is of said foreground part in said second macro-block is equal to or larger than a second 
predetermined value m (m<n), wherein said second macro-block is close to said first 
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macro-block where said first value is provided in said first macro-block approximation 
program code means; and 

program code means for outputting said segmentation mask. 

14. (Original) The computer readable medium as claimed in claim 13, further 
comprising: 

program code means for receiving each of third macro-blocks which has been 
determined as said background part; and 

program code means for providing said first value to said third macro-block when a 
difference image between a backgroimd image and an original image which correspond to 
said third macroblock includes a pixel which has a difference value equal to or larger than a 
threshold. 

15. (Original) A computer readable mediimi storing program code for causing a 
computer to extract a segmentation mask in object coding in moving image coding, 
comprising: 

program code means for receiving a foreground mask image; 

program code means for generating a number map by calculating the number of pixels 
of a foreground part for each of macro-blocks in said foreground mask image; 
program code means for initializing a foregroimd map; 

program code means for providing a predetermined value to each of positions in said 
foreground map corresponding to first macro-blocks when a value of said number map 
corresponding to said first macro-block is equal to or larger than a first predetermined value n 
(n^l); 

program code means for providing said predetermined value to each of positions in 
said foreground map corresponding to second macro-blocks when a value of said number 
map corresponding to said second macro-block is equal to or larger than a second 
predetermined value m (m<n), wherein said second macro-block is close to said first 
macroblock where said predetermined value is provided; and 

program code generating said segmentation mask from said foreground map and 
outputting said segmentation mask. 
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16. (Original) A segmentation mask extraction method for extracting a segmentation 
mask by using a difference image between a background image and an image, comprising the 
steps of: 

obtaining said difference image by calculating an absolute difference between said 
background image and said image fox each pixel; 

initializing an energy map for each macroblock of said difference image; 
calculating energy values for said each macro-block; 
obtaining an average of said energy values; 

calculating a foreground ratio which is a ratio of the size of a foreground ask to the 
size of said image; and 

generating said segmentation mask by using said foreground ratio. 

1 7. (Currently Amended) The segmentation mask exti ' action mctliod as claimed iu 
claun 16, fui - tlicr comprishiR Uic steps of: A segmentation mask extraction method for 
extractin g a segmentation mask by using a difference image between a background image and 
an image, comprising the steps of: 

obtaining said difference image by calculating an absolute difference between said 
background image and said image fox each pixel: 

initializing an energy man for each macroblock of said difference image: 
calculating energy values for said each macro-block: 
obtaining an average of said energy values: 

calculating a foregroimd ratio which is a ratio of the size of a foregroimd ask to the 
size of said image: 

generating said segmentation mask by using said foregroimd ratio: 

obtaining a divided value by dividing said energy value by said average for said each 
macroblock, and providing 0 as an energy value to a macro-block when said divided value is 
equal to or smaller than a (a ^ 1.0); 

obtaining a maximum energy value as a first predetermined value, setting a second 
predetermined value which is smaller than said first predetermined value, and initializing a 
foreground map; 

initializing a temporary foreground map; 
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providing a predetermine value to each macro-block position in said temporary 
foreground map where said energy value is equal to or larger than said first predetermined 
value; 

counting a count number of macro-blocks where said temporary foreground map has 
said predetermined value; 

generating said segmentation mask from said foreground map and outputting said 
segmentation mask if a value obtained by dividing said count mmiber by the number of all 
macro- locks is larger than a third predetermined value which is predetermined, if not, 
copying values of said temporary foreground map to said foreground map; 

iterating a providing step until a divided number obtained by dividing said coimt 
number by the number of all macro-blocks becomes larger than said third predetermined 
value, wherein said providing step is a step of providing said predetermined value to each 
macro-block position in said temporary foreground map where said energy value is equal to 
or larger than said second predetermined value, said each macro-block being close to a 
macro-block which has said predetermined value in said foreground map; and 

when said divided number does not become larger than said third predetermined value 
after iterating said providing step, copying values of said temporary foreground map to said 
foreground map, updating said first predetermined value and said second predetermined 
value, and performing said steps after said step of initializing said temporary foreground map. 

1 8. (Original) A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between background image and an image, 
comprising: 

means for obtaining said difference image by calculating an absolute difference 
between said background image and said image for each pixel; 

means for initializing an energy map for each macro-block of said difference image; 
means for calculating energy values for said each macro-block; 
means for obtaining an average of said energy values; 

means for calculating a foregroimd ratio which is a ratio of the size of a foreground 
mask to the size of said image; and 

means for generating said segmentation mask by using said foreground ratio. 

10 



AppL Sen No. 09/667,097 
Att. Docket No. 10746/21 

Reply to Final Office Action of June 2, 2003 

19. (Currently Amended) The segmentation mask exti - action mctliod as claim e d in 
claim 18. com p rising: A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between background image and an image, 
comprising: 

means for obtaining said difference image by calculating an absolute difference 
between said backgroimd image and said image for each pixel: 

means for initializing an energy map for each macro-block of said difference image: 
means for calculating energy values for said each macro-block: 
means for obtaining an average of said energy values: 

means for calculating a foreground ratio which is a ratio of the size of a foregroimd 
mask to the size of said image: 

means for generating said segmentation mask by using said foreground ratio: 

means for obtaining a divided value by dividing said energy value by said average for 
said each macro-block, and providing 0 as an energy value to a macro-block when said 
divided value is equal to or smaller than a (a 1 .0); 

means for obtaining a maximum energy value as a first predetermined value, setting a 
second predetermined value which is smaller than said first predetermined value, and 
initializing a foreground map; 

means for initializing a temporary foregroimd map; 

means for providing a predetermined value to each macro-block position in said 
temporary foreground map where said energy value is equal to or larger than said first 
predetermined value; 

means for counting a count nimiber of macro-blocks where said temporary foreground 
map has said predetermined value; 

means for generating said segmentation mask firom said foreground map and 
outputting said segmentation mask if a value obtained by dividing said count number by the 
number of all macro-blocks is larger than a third predetermined value which is 
predetermined, if not, copying values of said temporary foregroimd map to said foreground 
map; 

means for iterating a providing step until a divided number obtained by dividing said 
count number by the number of all macro-blocks becomes larger than said third 
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predetermined value, wherein said providing step is a step of providing said predetermined 
value to each macro-block position in said temporary foreground map where said energy 
value is equal to or larger than said second predetermined value, said each macro-block being 
close to a macro-block which has said predetermined value in said foregroxmd map; and 

means for copying values of said temporary foreground map to said foreground map, 
updating said first predetermined value and said second predetermined value, and performing 
said steps after said step of initializing said temporary foreground map, when said divided 
nixmber does not become larger than said third predetermined value after iterating said 
providing step. 

20. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask by using a difference image between a background 
image and an image, comprising: 

program code means for obtaining said difference image by calculating an absolute 
difference between said background image and said image for each pixel; 

program code means for initializing an energy map for each macro-block of said 
difference image; 

program code means for calculating energy values for said each macro-block; 
program code means for obtaining an average of said energy values; 
program code means for calculating a foreground ratio which is a ratio of the size of a 
foreground mask to the size of said image; and 

program code means for generating said segmentation mask by using said foregroxmd 

ratio. 

21 . (Currently Amended) Tlic computer readable medium as claimed in claim 20, 
compii&iim. A computer readable medium storing program code for causing a computer to 
extract a s egmentation mask bv using a difference image between a background image and an 
image, comprising: 

program code means for obtaining said difference image bv calculating an absolute 
difference between said background image and said image for each pixel: 
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program code means for initializing an energy map for each macro-block of said 
difference image: 

program code means for calculating energy values for said each macro-block: 
program code means for obtaining an average of said energy values: 
program code means for calculating a foreground ratio which is a ratio of the size of a 
foreground mask to the size of said image: 

program code means for generating said segmentation mask bv using said foreground 

ratio: 

program code means for obtaining a divided value by dividing said energy value, by 
said average for said each macro-block, and providing 0 as an energy value to a macro-block 
when said divided value is equal to or smaller than a (a ^ 1 .0); 

program code means for obtaining a maximum energy value as a first predetermined 
value, setting a second predetermined value which is smaller than said first predetermined 
value, an initializing a foreground map; 

program code means for initializing a temporary foregroimd map; 

program code means for providing a predetermined value to each macro-block 
position in said temporary foreground map whey said energy value is equal to or larger than 
said first predetermined value; 

program code means for counting a count number of macro-blocks where said 
temporary foreground map has said predetermined value; 

program code means for generating said segmentation mask fi-om said foreground 
map and outputting said segmentation mask if a value obtained by dividing said count 
mmiber by the number of all macro-blocks is larger than a third predetermined value which is 
predetermined, if not, copying values of said temporary foreground map to said foregroimd 
map; 

program code means for iterating a providing step until a divided number obtained by 
dividing said count number by the umber of all macro-blocks becomes larger than said third 
predetemiined value, wherein said providing step is a step of providing said predetermined 
value to each macro-block position in said temporary foreground map where said energy 
value is equal to or larger than said second predetermined value, said each macro-block being 
close to a macro block which has said predetermined value in said foreground map; and 
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program code means for copying values of said temporary foreground map to said 
foreground map, updating said first predetermined value and said second predetermined 
value, and performing said steps after said step of initializing said temporary foreground map, 
when said divided umber does not become larger than said third predetermined value after 
iterating said providing step. 

22. (Original) A segmentation mask extraction method for extracting a segmentation 
mask by using a difference image between a background image and an image, comprising: 

a first step of regarding each of first macro-blocks as the foreground when an energy 
value of said first macro-block which is obtained by said difference image is equal to or 
larger than a first predetermined value; 

a second step of regarding each of second macro-blocks as the foregroimd when an 
energy value of said second macro-block is equal to or larger than a second predetermined 
value, said second macro-block being close to a macro-block which is determined as the 
foreground in said first step. 

23. (Original) The segmentation mask extraction method as claimed in claim 22, 
fiuther comprising a step of iterating said second step for predetermined times. 

24. (Original) A segmentation mask extraction method for extracting a segmentation 
mask by using a difference image between a background image and an image, comprising the 
steps of: 

calculating energy value of each macro-block fi-om said difference image and 
calculating an average of said energy values; 

obtaining a divided value by dividing said energy value by said average for said each 
macro-block, and providing 0 as an energy value to a macro-block when said divided value is 
equal to or smaller than a predetermined value; 

regarding each of first macro-blocks as the foreground when said energy value of said 
first macro-block is equal to or larger than a first predetermined value; 

iterating, predetermined times, a step of regarding each of second macro-blocks as the 
foreground when said energy value of said second macro-block is equal to or larger than a 
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second predetermined value, said second macro-block being close to said first macro-block 
which is determined as the foreground. 

25. (Original) A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between a background image and an image, 
comprising: 

first means for regarding each of first macro-blocks as the foreground when an energy 
value of said first macro-block which is obtained by said difference image is equal to or 
larger than a first predetermined value; 

second means for regarding each of second macro-blocks as the foreground when an 
energy value of said second macro-block is equal to or larger than a second predetermined 
value, said second macro-block being close to a macro-block which is determined as the 
foreground in said first means. 

26. (Original) The segmentation mask extraction apparatus as claimed in claim 25, 
further comprising means for iterating the process by said second means for predetermined 
times. 

27. (Original) A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between a background image and an image, 
comprising: 

means for calculating energy values of each macro-block firom said difference image 
and calculating an average of said energy values; 

means for obtaining a divided value by dividing said energy value by said average for 
said each macro-block, and providing 0 as said energy value to a macro-block when said 
divided value is equal to or smaller than a predetermined value; means for regarding each of 
first macro-blocks as the foreground when said energy value of said first macro-block is equal 
to or larger than a first predetermined value; 

means for iterating, predetermined times, a step of regarding each of second 
macro-blocks as the foreground when said energy value of said second macro-block is equal 
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to or larger than a second predetermined value, said second macro-block being close to said 
first macro-block which is determined as the foreground. 

28. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask by using a difference image between a backgroimd 
image and an image, comprising: 

first program code means for regarding each of first macro-blocks as the foreground 
when an energy value of said first macro-block which is obtained by said difference image is 
equal to or larger than a first predetermined value; 

second program code mean for regarding each of second macro-blocks as the 
foregroxmd when an energy value of said second macro-block is equal to or larger than a 
second predetermined value, said second macro-block being close to macro-block which is 
determined as the foreground in said first program code means. 

29. (Original) The computer readable mediimi as claimed in claim 28, fiirther 
comprising program code means for iterating the process by said second program code means 
for predetermined times. 

30. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask by using a difference image between a backgroimd 
image and an image, comprising: 

program code means for calculating energy values of each macro-block fi-om said 
difference image and calculating an average of said energy values; 

program code means for obtaining a divided value by dividing said energy value by 
said average for said each macro-block, and providing 0 as said energy value to a 
macro-block when said divided value is equal to or smaller than predetermined value; 

program code means for regarding each of first macro-blocks as the foreground when 
said energy value of said first macro-block is equal to or larger than a first predetermined 
value; 

program code means for iterating, predetermined times, a step of regarding each of 
second macro-blocks as the foregroimd when said energy value of said second macro-block is 

16 




r 

Appl. Ser. No. 09/667,097 
Art. Docket No. 10746/21 

Reply to Final Office Action of June 2, 2003 

equal to or larger than a second predetermined value, said second macro-block being close to 
said first macro-block which is determined as the foreground. 
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